Use SQL null semantics for DuckDB IN and NOT IN pushdown by robert3005 · Pull Request #10054 · vortex-data/vortex

robert3005 · 2026-09-25T13:56:23Z

Summary

DuckDB membership pushdown used Vortex's default null rules, which can make a nonmatch in NOT IN (1, NULL) true. Use SQL null semantics so that result remains unknown and the row is excluded.

Stacked on #10053; uses SQL null semantics from #10057 and membership optimizations from #10052.

Changes

Convert both bound IN/NOT IN expressions and IN table filters through in_list.
Leave membership lists containing nonconstant expressions in DuckDB instead of advertising support and then failing conversion.
Add result and EXPLAIN regressions for null elements in either position, nullable needles, negation, scans through views, and a 1,000-element set. Add a nonconstant-list fallback regression.

Validation: cargo clippy -p vortex-duckdb --all-targets --all-features -- -D warnings passed on this branch. Runtime tests were not executed.

codspeed · 2026-09-25T14:03:43Z

Merging this PR will regress 11 benchmarks

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️

1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

bench_compare_primitive[(10000, 2)]

⚠️

13 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

random_i128[0.01]

pcodec_compress_f64

decompress[u32, (1000, 256)]

decompress[u16, (4000, 256)]

filter_run_end[len=4096_run=16_density=0.1]

multipolygon_mixed_validity

density_sweep_dense_runs[0.001]

take_varbin

zstd_compress_string

ilike_contains

…and 3 more

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 7 improved benchmarks
❌ 11 regressed benchmarks
✅ 2201 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`optimize_lookup_predicate[ids=192, shape=in_list]`	51 µs	75.5 µs	-32.41%
❌	Simulation	`optimize_lookup_predicate[ids=128, shape=in_list]`	51.6 µs	72.9 µs	-29.18%
❌	Simulation	`optimize_lookup_predicate[ids=64, shape=in_list]`	51.5 µs	70.3 µs	-26.69%
❌	Simulation	`optimize_lookup_predicate[ids=256, shape=in_list]`	79.6 µs	105.2 µs	-24.34%
❌	Simulation	`optimize_lookup_predicate[ids=16, shape=in_list]`	52.4 µs	69.1 µs	-24.15%
❌	Simulation	`take_map[(0.1, 1.0)]`	226.1 µs	292.1 µs	-22.6%
❌	Simulation	`optimize_lookup_predicate[ids=1, shape=in_list]`	75.3 µs	89.7 µs	-16.13%
❌ ⚠️	Simulation	`random_i128[0.01]`	7.2 µs	8.5 µs	-15.01%
❌	Simulation	`sum_i64`	194.4 µs	225.1 µs	-13.62%
❌	Simulation	`sum_v2_i64`	194.2 µs	224.8 µs	-13.58%
❌	WallTime	`filter_avx2[64]`	2.1 µs	2.3 µs	-10.64%
⚡	Simulation	`i64_random_chunked[256]`	13,907.4 µs	225.7 µs	×62
⚡	Simulation	`utf8_random[256]`	12,973.6 µs	415.1 µs	×31
⚡	Simulation	`i64_random[256]`	3,928.8 µs	169.2 µs	×23
⚡	Simulation	`nested_list_random[32]`	6.7 ms	1 ms	×6.4
⚡	WallTime	`filter_avx512[80]`	3.2 µs	2.3 µs	+42.57%
⚡	WallTime	`filter_avx512[64]`	2.9 µs	2.2 µs	+33.24%
⚡	WallTime	`bitpack_blocked_compress_avx512`	5.6 µs	4.8 µs	+17.07%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing rk/duckdb-list-contains (461e8a7) with develop (751e0af)¹}

No successful run was found on develop (b085f29) during the generation of this report, so 751e0af was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

robert3005 · 2026-09-28T22:18:03Z

+            ) {
+                let mut children = op.children();
+                return children.next().is_some_and(can_push_expression)
+                    && children.all(|child| matches!(child.as_class(), Some(BoundConstant(_))));


this seems stricter than before

robert3005 · 2026-09-28T22:18:46Z

        TableFilterClass::InFilter(values) => {
            // TODO(ngates): I'm pretty sure we actually need this as ScalarValue with the
            //  scope dtype
            let scalars: Vec<_> = values.iter().map(Scalar::try_from).try_collect()?;


we should remove this

Convert both IN table filters and bound membership expressions through in_list, so a null set element leaves nonmatches unknown. Decline nonconstant lists instead of promising conversion and failing later. Add scan-result and EXPLAIN regressions for nullable sets, views, large constant lists, and fallback for nonconstant list elements. Signed-off-by: Robert Kruszewski <github@robertk.io>

Signed-off-by: Robert Kruszewski <github@robertk.io>

robert3005 added this pull request to stack #10055 September 25, 2026 13:56

This was referenced Sep 25, 2026

Optimize list_contains with prepared constant sets #10052

Merged

Expose list membership with SQL null semantics in Java and Python #10053

Open

robert3005 removed this pull request from stack #10055 September 25, 2026 14:05

robert3005 force-pushed the rk/list-contains-bindings branch from 2aaa1b6 to 1b63fd8 Compare September 25, 2026 14:06

robert3005 force-pushed the rk/duckdb-list-contains branch from 7f7f687 to 29fa2b0 Compare September 25, 2026 14:06

robert3005 added this pull request to stack #10058 September 25, 2026 14:06

robert3005 mentioned this pull request Sep 25, 2026

Add SQL null semantics to list_contains #10057

Merged

robert3005 removed this pull request from stack #10058 September 25, 2026 14:09

robert3005 force-pushed the rk/list-contains-bindings branch from 1b63fd8 to 3151b45 Compare September 25, 2026 14:09

robert3005 added this pull request to stack #10061 September 25, 2026 14:09

robert3005 force-pushed the rk/duckdb-list-contains branch from 29fa2b0 to adc15ee Compare September 25, 2026 14:09

robert3005 mentioned this pull request Sep 25, 2026

Add list_contains membership benchmarks #10060

Merged

robert3005 force-pushed the rk/duckdb-list-contains branch from adc15ee to 2535299 Compare September 25, 2026 15:53

robert3005 force-pushed the rk/list-contains-bindings branch from 3151b45 to b6ffdda Compare September 25, 2026 15:53

robert3005 force-pushed the rk/duckdb-list-contains branch from 2535299 to cafd7d2 Compare September 25, 2026 16:28

robert3005 force-pushed the rk/list-contains-bindings branch 2 times, most recently from ed99a18 to 0e3443a Compare September 25, 2026 17:10

robert3005 force-pushed the rk/duckdb-list-contains branch 2 times, most recently from 04bd414 to 459162a Compare September 25, 2026 17:26

robert3005 force-pushed the rk/list-contains-bindings branch 2 times, most recently from 2242462 to 828a8f7 Compare September 25, 2026 20:28

robert3005 force-pushed the rk/duckdb-list-contains branch 2 times, most recently from c0e6065 to 843d57c Compare September 28, 2026 20:43

robert3005 force-pushed the rk/list-contains-bindings branch from 828a8f7 to 3dd8f74 Compare September 28, 2026 20:43

robert3005 commented Sep 28, 2026

View reviewed changes

robert3005 force-pushed the rk/duckdb-list-contains branch from 843d57c to e4b6598 Compare September 29, 2026 19:13

robert3005 force-pushed the rk/list-contains-bindings branch from 3dd8f74 to 2a51612 Compare September 29, 2026 19:13

robert3005 removed this pull request from stack #10061 October 12, 2026 10:29

robert3005 force-pushed the rk/list-contains-bindings branch from 2a51612 to 446956a Compare October 12, 2026 10:29

robert3005 marked this pull request as ready for review October 12, 2026 10:29

robert3005 force-pushed the rk/duckdb-list-contains branch from e4b6598 to 2a977a1 Compare October 12, 2026 10:29

robert3005 added this pull request to stack #10441 October 12, 2026 10:29

robert3005 marked this pull request as draft October 12, 2026 10:29

robert3005 removed this pull request from stack #10441 October 12, 2026 10:29

robert3005 added this pull request to stack #10443 October 12, 2026 10:29

jackylee-ch mentioned this pull request Oct 14, 2026

Support Spark floating point semantics in bound comparisons #10064

Open

robert3005 force-pushed the rk/duckdb-list-contains branch from 2a977a1 to 5b12a0f Compare October 14, 2026 17:32

robert3005 force-pushed the rk/list-contains-bindings branch from 446956a to 6801f5b Compare October 14, 2026 17:32

robert3005 force-pushed the rk/duckdb-list-contains branch from 5b12a0f to 757e4da Compare October 14, 2026 18:26

robert3005 added 3 commits October 14, 2026 19:34

Remove unused DuckDB conversion import

3cc7ad2

Signed-off-by: Robert Kruszewski <github@robertk.io>

Format DuckDB membership pushdown test

461e8a7

Signed-off-by: Robert Kruszewski <github@robertk.io>

robert3005 force-pushed the rk/duckdb-list-contains branch from 757e4da to 461e8a7 Compare October 14, 2026 18:34

robert3005 force-pushed the rk/list-contains-bindings branch from 856b420 to 2bda1b5 Compare October 14, 2026 18:34

robert3005 removed this pull request from stack #10443 October 14, 2026 18:35

robert3005 changed the base branch from rk/list-contains-bindings to develop October 14, 2026 18:35

Navigation Menu

Repository navigation

Use SQL null semantics for DuckDB IN and NOT IN pushdown - #10054

Use SQL null semantics for DuckDB IN and NOT IN pushdown#10054
robert3005 wants to merge 3 commits into
developfrom
rk/duckdb-list-contains

robert3005 commented Sep 25, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Sep 25, 2026 •

edited

Loading

Uh oh!

robert3005 Sep 28, 2026

Uh oh!

robert3005 Sep 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

robert3005 commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

codspeed Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 11 benchmarks

Performance Changes

Footnotes

Uh oh!

robert3005 Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

robert3005 commented Sep 25, 2026 •

edited

Loading

codspeed Bot commented Sep 25, 2026 •

edited

Loading